Image Based Visual Simulation and Tele-Assisted Robot Control

In tele-assistance a robot operates partly guided by a human and partly autonomously. Tele-assistance thus fills in the gap between full autonomy, and direct control by a human tele-operator. Instead of directly controlling the motor actions of a robot, in tele-assistance the human interacts with the robot using a deictic high level language. Pook and Ballard have implemented such a system using a hand sign language, sensed by an exoskeleton[26]. In this work we present a method in which the robot is instructed through visual pointing. The human operator instructs the robot by pointing out (currently using a mouse and images displayed on a computer screen) desired manipulations. The tele-assistance system transforms the coarse task descriptions supplied by the human into a visual (camera) space task and trajectory plan. The manipulation is carried out using visual servoing on an uncalibrated hand-eye system. For this we have developed an integrated model estimation, control and visual simulation system capable of transforming these visual space plans into motor action and providing image based synthethic visual feedback to the human operator in order to compensate for the delayed response of the real robotic system.